\section{Lemona}

\subsection{Background}

\subsubsection{System Calls}

In computer systems, the \emph{kernel} is the main component of the operating
system. It is the piece of software in charge of managing the system
resources (i.e. communicating with the hardware devices). Being the
lowest abstraction layer between software and hardware, every
operation needing support from the underlying hardware or requesting
to update the hardware's state will go through it.

Thus, applications that need to access the hardware use the \emph{kernel}
layer interface: the \emph{system calls} (commonly called \emph{syscalls}). \emph{Syscalls}
are functions invoked by user applications to request some service or
resource from the \emph{kernel} \cite{love:LINUXSYSPROG}.

With this in mind, it is obvious that it is possible to have a
complete monitoring of what applications request to do with the
resources of the computer system by adding logging mechanisms at the
entrance and exit of these functions.

\subsubsection{Scheduler}

We have seen that we can monitor access to the computer system's
resources by simply logging the usage of the \emph{kernel}
\emph{syscalls}. However, due to the nature of the \emph{kernel}, we need to
capture log from other parts of the \emph{kernel} to be able to draw a
timeline of an application's execution, namely the ones in charge of
allocating time slices to applications: the \emph{scheduler}. The
\emph{scheduler} is the part of the \emph{kernel} deciding who can access the
CPU at a given moment in time and for how long. While a system gives
the illusion of having several applications running concurrently -
multi-processor systems aside - actually only one single process can
make use of the CPU at any given moment. The \emph{scheduler} allocates a
tiny amount of time to each application which leads to this illusion
of concurrency \cite{bovet:ULK}.

\subsubsection{Memory Mapping}

Monitoring \emph{syscalls} permits to see easily every change made to files
on disk. However, modern \emph{kernel}s give the possibility to an
application to map files in memory by using the mmap \emph{syscall}. This
allows an application to access directly the content of a file as if
it was part of its own memory space (i.e. without having to issue the
corresponding \emph{syscall} when it needs to read or write to it). It also
permits to reduce the performance impact on data manipulation and
moves by avoiding the intermediate copy of the data in a \emph{kernel}
buffer. Every time the application will try to access a part of the
file not yet mapped into memory, the system will emit a so-called Page
Fault. This event will suspend the application execution and inform
the \emph{kernel} that the application tried to read or write a yet unmapped
position. Then the \emph{kernel} will load the relevant data into memory and
allow the application to resume its execution, starting over from the
point preceding the \emph{Page Fault}. The application can now read or write
from or to the portion of the file as intended. In order to log the
data modified and accessed by this kind of application, it is thus
necessary to alter the part of the \emph{kernel} in charge of handling Page
Faults in addition to the logging of the mmap related \emph{syscalls}.

Because these file portions are loaded using the granularity of a
so-called \emph{Page Size} (usually 4 Kilobytes), the logging mechanism is
bound to use the same. This is translated by the fact that the logging
of the changes made to a portion of the file can only be done when the
application is removed from the CPU by the \emph{scheduler}. Doing so earlier
would not guarantee that every changes made to a given portion (page)
of the file will be logged.

\subsection{Architecture}

\begin{figure*}
  \centering
  \includegraphics[scale=0.60]{images/architecture/lemona-architecture-5.png}
  \caption{Lemona's Architecture}
  \label{fig:fig4}
\end{figure*}

\textbf{Lemona} is a compound of several components (Figure \ref{fig:fig3} and Figure \ref{fig:fig4}), separated in three categories:

\begin{figure*}
  \centering
  \includegraphics[scale=0.55]{images/architecture/lemona-architecture-components.png}
  \caption{Lemona's Components}
  \label{fig:fig5}
\end{figure*}

\subsubsection{Monitoring Components}

\paragraph{Kernel Patches}

These are alterations to the Linux Kernel codebase needed to monitor
\emph{syscalls} entry and exit points, as well as application scheduling and
memory mapped files related \emph{Page Faults}. The patches are designed to
allow a literate user to remove the \textbf{Lemona} functionalities (and thus
their processing overhead and memory footprint) from the \emph{kernel} if he
chooses to disable it upon compilation.

\paragraph{Loadable Kernel Modules}

While the patches add code at key \emph{kernel} points for logging, the
driver does the actual logging. It structures the logged information
and sends the result to the various designated backends. Even Though
we made it possible to dynamically load and unload the \textbf{Lemona}
functionalities, we only did so to facilitate debugging and initial
testing, or for end-users convenience. However, administrators of
production systems should build the module statically for security
reasons. This will for instance avoid early logs to be lost.

So far, the modules can only transmit the generated logs via two
different methods:

\begin{itemize}
  \item Via an (un)encrypted socket
  \item Via the kernel's relay functionality
\end{itemize}

It is important to note that data can easily be lost using any of
these techniques. In the first case, if the server does not answer or
is too slow to treat the data, incoming log data will be simply
drop. The same applies to the relay facility. Once the buffers are
full, new incoming logs will be discarded until the user application
processes enough data to free up one or more buffers.

\subparagraph{Socket Method}

Upon startup, the module will prompt the user for the passphrase used
to encrypt sent to the logging server. Alternatively, this passphrase
could be stored on the local file system for automatic startup; but
this increase the security risk. If the link to the server is
considered sure enough, it is possible to disable the encryption
mechanism and thus reduce the CPU consumption impact of
\textbf{Lemona}.

\subparagraph{Kernel Relay Method}

This method exists mainly for testing purposes. Using the relay
functionality of the \emph{kernel}, the logs are made available to
user space applications using the debugfs file system. User
applications are then free to transfer the data back to a logging
server or on a local database.

\subsubsection{Logging Components}

\paragraph{Databases Servers}

SQL databases are used to stock the logs, making it easy for forensics
expert to query them and request specific data. This allows for quick
access to relevant informational datasets, and for easy and flexible
narrowing of the search domain (e.g. by date, applications,
etc...). Depending on the database server used on the backend, useful
requests could be recorded for future reference. In addition, we plan
to rely heavily on database caching mechanisms to speed up information
retrieval.

\paragraph{Logging Servers}

This application is in charge of receiving the logs from the
\emph{kernel}. Upon startup, the \textbf{Lemona} driver will connect
to the logging server and start sending him data as it comes
through. In order to maximize throughput, the server should preferably
be directly connected to the monitoring machine via a fast link
(i.e. Gigabit Ethernet). For security reasons, the machine hosting the
server should be hardened, and avoid unnecessary services should be
disabled and uninstalled altogether.

The server application is actually a compound of two programs. The
first one is in charge of receiving the data sent by the driver. It
immediately appends the data to a set of buffer files without any
modification. Once a file is full, another one is created. The second
application will read already filled files, decrypt them and insert
the data in the database. This architecture avoids heavy slowdowns
upon data reception by delegating the load of the decryption (if any)
on the receiving application.

\subsubsection{Forensics Component}

\paragraph{Database-Querying Applications}

We are currently working on a querying application to provide simple
interactive access to the database data. This application should
permit the creation of specific queries without the user having to
know the database schema. It would present the output in a meaningful
way targeted at forensics expert.

This part of the \textbf{Lemona} architecture is still under heavy R\&D though.


\subsection{Experimentation}

\subsubsection{Test Configuration}

The results have been gathered using the following configuration:

\begin{itemize}
  \item \textbf{CPU:} Intel(R) Pentium(R) M processor 1.80GHz
  \item \textbf{RAM:} 1GB
  \item \textbf{OS:} GNU/Linux Gentoo
  \begin{itemize}
    \item \textbf{CFLAGS:} -O2 -march=pentium-m -pipe -fomit-frame-pointer
  \end{itemize}
  \item \textbf{VM:} VirtualBox-2.0.4
  \begin{itemize}
    \item \textbf{Guest OS:} Linux Gentoo
    \item \textbf{Base Memory:} 256MB
    \item \textbf{Video Memory:} 16MB
    \item \textbf{Network:} PCnet-FAST III (host interface, tap0)
  \end{itemize}
\end{itemize}

There are two different batteries of tests to demonstrate the capabilities
of the \textbf{Lemona} system:

\begin{itemize}
  \item Performance \& Availability Benchmarks
  \item Functional Tests
\end{itemize}


All tests have been carried using a statically compiled lemona module
using different configurations:

\begin{itemize}
  \item lemona + net

  The network module is enabled, but there is no server to send data to.

  \item lemona + net + basket

  The network module is enabled and the \emph{basket} server is
  accepting data.

  \item lemona + relay

  The \emph{kernel}'s relay facility is enabled, but no program is
  reading the output.

  \item lemona + relay

  The \emph{kernel}'s relay facility is enabled and the output is read
  by cat to \emph{/dev/null}.
\end{itemize}


\subsubsection{Performance \& Availability Benchmarks}

These benchmarks aim to demonstrate the usability of \textbf{Lemona} in a
test environment, by proving that its performance impact does not render
the system unavailable.

We want to check the performance impact of the \textbf{Lemona} system for
"standard" system usage, based on tests presented by \textbf{Forensix} \cite{goel:FORENSIX}:

\begin{itemize}
  \item benchmarking the build of a Linux \emph{kernel} from scratch
  \item benchmarking the throughput of an \textbf{Apache Web Server}.
\end{itemize}

Each benchmark set is carried using 12 different configurations:

\begin{itemize}
  \item \textbf{Lemona} has neither been built, nor loaded
  \item \textbf{Lemona} has been built as a module, but the module is not loaded
  \item \textbf{Lemona} has been built as a module and the module has been loaded:
  \begin{itemize}
    \item Relay reporting is enabled / Socket reporting is disabled
    \item Relay reporting is disabled / Socket reporting is enabled with encryption
    \item Relay reporting is disabled / Socket reporting is enabled without encryption
    \item Relay and Socket reporting are enabled (with encryption)
    \item Relay and Socket reporting are enabled (without encryption)
  \end{itemize}
  \item \textbf{Lemona} has been built statically:
  \begin{itemize}
    \item Relay reporting is enabled / Socket reporting is disabled
    \item Relay reporting is disabled / Socket reporting is enabled with encryption
    \item Relay reporting is disabled / Socket reporting is enabled without encryption
    \item Relay and Socket reporting are enabled (with encryption)
    \item Relay and Socket reporting are enabled (without encryption)
  \end{itemize}
\end{itemize}

Recorded information for the performance benchmarks include:

\begin{itemize}
  \item timestamps for:
  \begin{itemize}
    \item benchmark start
    \item benchmark end
    \item each of the following readings
  \end{itemize}
  \item regular and peak readings for:
  \begin{itemize}
    \item I/O throughput levels (in bits per seconds)
    \item CPU consumption levels (in percentage of available processing power)
    \item Memory consumption levels (in bytes and percentage of available memory)
    \item database inputs (in records per minutes)
    \item average and peak log file sizes
    \item average and total number of database entries
  \end{itemize}
\end{itemize}

\subsubsection{Functional Tests}

These tests aim to demonstrate the functional feasibility of \textbf{Lemona}
within a real-life environment. Using real case scenarios (yet to be
determined), the application will be tested to assert:

\begin{itemize}
  \item \textbf{Lemona}'s capacity at collecting relevant information

  Information relative to security breaches or system failures should
  be recorded on the host targeted and stored in the database to be
  identified easily for later forensics research.

  \item \textbf{Lemona}'s capacity at retrieving relevant information

  The information collected during the first phase of the test should
  be easily and quickly accessed using appropriate queries.
\end{itemize}

Recorded information for the functional tests include:

\begin{itemize}
  \item timestamps for:
  \begin{itemize}
    \item benchmark start
    \item benchmark end
    \item each of the following actions upon start and completion
  \end{itemize}
  \item readings for:
  \begin{itemize}
    \item returned noise and false positives (absolute and relative reading)
    \item lookup time (in seconds)
  \end{itemize}
\end{itemize}

\subsection{Results}

\subsubsection{Notes on the \textbf{Lemona} Results}

As of this writing, results are still being collected. We are just
providing below some insights on our expectations and our
checkpoints, as well as the current results we managed to collect.

Currently, we have not collected the results for the functional tests
outlined above, as \textbf{Lemona} is not feature complete from the computer
forensics point of view, and we did not perform tests with an active
encryption layer, as it is not yet part of our proof of concept.

Regarding the benchmark comparison with \textbf{Forensix}, we would like to
emphasize that they might not be easy to compare with a 1-to-1 ratio,
for the following reasons:

\begin{itemize}
  \item \textbf{Coverage}

  \textbf{Forensix} does not record every single operation, and only
  focuses on specific processes, whereas \textbf{Lemona} aims at
  monitoring the complete system activity. On the other hand,
  \textbf{Lemona} does not (yet) monitor all the \emph{system calls};
  but the ones implemented are monitored for all processes.

  \item \textbf{Hardware and Software Configurations}

  We do not have at our disposal the hardware resources to perform
  tests matching these of \textbf{Forensix}. Therefore, we use
  different hardware and system configurations, and our tests were run
  within a \emph{virtual machine}.

  \item \textbf{Tests Replicability}

  \textbf{Lemona} and \textbf{Forensix} are compared based on similar
  actions, but with different inputs. For instance, they used
  different kernel sources for the performance test, which means that
  the kernel compilation time might not be as relevant.

\end{itemize}

However, we believe these benchmarks provide a valuable insight and
indication of \textbf{Lemona}'s usability on a real system, and that the
transparence we provide with this comparison \textbf{Forensix} might allow
readers to determine if our solution matches their needs.

\subsubsection{Coverage}

We intend to cover close to 100\% of a system's activity, in that we
actually trace all existing \emph{system calls} and monitor memory mapped
areas' read and write accesses. We should therefore be able to monitor
all actions executed by all (human and machine) users logged onto a
system as the system's core activity. We expect to be able to collect
enough data to virtually reconstruct a complete system from a given
checkpoint.

Our proof of concept achieves a technical validation of this
objective, by monitoring twenty-two (22) \emph{system calls}, among
which some of the most intensive ones, performing I/O operations.

\subsubsection{Performance}

\subsubsection{CPU: Linux Kernel 2.6.26.3 Compilation}

The kernel has been compiled using the configuration file found in
\emph{arch/x86/configs/i386\_defconfig\_} that comes along with its
sources.

Between each test:

\begin{itemize}
  \item the virtual machine has been reset
  \item the 'old' Linux directory has been deleted and copied over again.
\end{itemize}

For each test \emph{make menuconfig} has been executed before launching the
compilation. No modification has been made to the default
configuration, we simply have him save the \emph{.config} file.

We use the \emph{time make} command to produce the following measurements.

\begin{table*}
  \renewcommand{\arraystretch}{1.3}
  \caption{Performance Results}
  \label{table:performance}
  \centering
  \begin{tabular}{ | l | l | l | l | l | }
    \hline
    \textbf{Test}                     & \textbf{User Time} & \textbf{System Time} & \textbf{Real Time} & \textbf{CPU}  \\ \hline
    \textbf{w/o lemona}               & 283s               & 226s                 & 519s              & 98\%           \\ \hline
    \textbf{w/ lemona + net}          & 242s               & 256s                 & 531s  (+02.31\%)  & 93\%           \\ \hline
    \textbf{w/ lemona + net + basket} & 269s               & 334s                 & 670s  (+29.09\%)  & 90\%           \\ \hline
    \textbf{w/ lemona + relay}        & 300s               & 292s                 & 604s  (+16.37\%)  & 98\%           \\ \hline
    \textbf{w/ lemona + relay + cat}  & 247s               & 256s                 & 1028s (+98.07\%)  & 49\%           \\ \hline
    \hline
  \end{tabular}
\end{table*}

As we predicted, we notice a significant impact on the system's
performance when \textbf{Lemona} is enabled. However, and even though
\textbf{Lemona} performs worse than \textbf{Forensix} (See Appendix
A), our proof of concept remains usable inspite of the overhead.

\subsubsection{Network: Apache 2 Benchmark}

An \textbf{Apache 2} server has been installed, default configuration has
been kept, and a 169 bytes \emph{index.html} file created.

The benchmark has been done using the 'ab' tool from the \textbf{Apache
Foundation} as follows:

\begin{verbatim}
ab -n 10000 -c 10 http://10.0.42.2/
\end{verbatim}

\begin{table*}
  \renewcommand{\arraystretch}{1.3}
  \caption{Network Results}
  \label{table:network}
  \centering
  \begin{tabular}{ | l | l | l | l | }
    \hline
    \textbf{Test}                     & \textbf{Total Time} & \textbf{Requests/s} & \textbf{Real Time}   \\ \hline
    \textbf{w/o lemona}               & 5.767s              & 1733.97             & 687.49Kb             \\ \hline
    \textbf{w/ lemona + net}          & 5.977s (+03.64\%)   & 1673.00 (-03.51\%)  & 663.32Kb (-03.51\%)  \\ \hline
    \textbf{w/ lemona + net + basket} & 6.912s (+19.85\%)   & 1446.73 (-16.56\%)  & 573.61Kb (-16.56\%)  \\ \hline
    \textbf{w/ lemona + relay}        & 6.091s (+05.61\%)   & 1641.63 (-05.32\%)  & 650.88Kb (-05.32\%)  \\ \hline
    \textbf{w/ lemona + relay + cat}  & 6.700s (+16.17\%)   & 1492.50 (-13.92\%)  & 591.75Kb (-13.92\%)  \\ \hline
    \hline
  \end{tabular}
\end{table*}

As for the CPU consumption, the network resources take a significant
hit if \textbf{Lemona} is configured to transmit its traces over the
network. Considering \textbf{Lemona} will be transmitting all the traces over a
network interface, it will be under considerable load. However, we
might implement buffering techniques to improve this, and we think the
solution should provide satisfactory results to be used on a
\emph{LAN}. Benchmarks will show if this is a viable solution for systems
residing on more distant networks, such as \emph{WAN}/\emph{VPNs}.

\paragraph{Memory}

As of today, the tracing structures' memory footprint is less than 30
bytes, if they do not require additional parameters. Regarding the
memory consumption on the storage point's side, which we expect to
have a high rate, we are not able at the moment to provide valuable
benchmarks, for various reasons, which are explained below.

\begin{itemize}
  \item \textbf{Lemona}'s internal storage structures are morphing because:
  \begin{itemize}
    \item We are still in the process of modifying our API;
    \item They depend on which system call is being traced.
  \end{itemize}
  \item The rate at which the traces will be generated is really variable because:
  \begin{itemize}
    \item It depends on the kind of load the monitored host is being put under;
    \item It depends on the kind of activity the monitored host is performing (watching a movie or doing a full-text search will not have the same system call throughputs)
  \end{itemize}
\end{itemize}
